Ideograms for Captured Expressions

ABSTRACT

Particular embodiments of a method comprise analyzing an image to classify an expression displayed on a face shown in a captured image. Image analysis may include detecting the face in the image, generating a characterization of features of the face, and classifying the expression based on the characterization. The characterization of facial features may be based on benchmark metrics for a particular expression. One or more ideograms (e.g., written characters, symbols or images that represent an idea or thing) may be selected based on the expression. The selection may be based on one or more match scores for the expression. The match scores may be determined based on an ideogram dictionary or an ideogram usage history. The selected ideograms may then be presented, in association with the face, on a screen for the computing device.

TECHNICAL FIELD

This disclosure generally relates to presentation of content on acomputing device.

BACKGROUND

A social-networking system, which may include a social-networkingwebsite, may enable its users (such as persons or organizations) tointeract with it and with each other through it. The social-networkingsystem may, with input from a user, create and store in thesocial-networking system a user profile associated with the user. Theuser profile may include demographic information, communication-channelinformation, and information on personal interests of the user. Thesocial-networking system may also, with input from a user, create andstore a record of relationships of the user with other users of thesocial-networking system, as well as provide services (e.g., wall posts,photo-sharing, event organization, messaging, games, or advertisements)to facilitate social interaction between or among users.

The social-networking system may send over one or more networks contentor messages related to its services to a mobile or other computingdevice of a user. A user may also install software applications on amobile or other computing device of the user for accessing a userprofile of the user and other data within the social-networking system.The social-networking system may generate a personalized set of contentobjects to display to a user, such as a newsfeed of aggregated storiesof other users connected to the user.

SUMMARY OF PARTICULAR EMBODIMENTS

Particular embodiments described in this disclosure relate to the use ofideograms in a text message input by an authoring user on a computingdevice. The author may input text on a mobile device to be sent to otherusers through a number of methods, including for example a messagingplatform or a social-networking system. The ideograms will be sent toother users along with the message, and the receiving users will be ableto view the ideogram in conjunction with the text message in the samerelative layout as the author's created message.

In one particular embodiment, one or more of the ideograms that may beplaced in a message may have restrictions on use or viewing. A user ofthe messaging platform or social-networking system may need to beauthorized in order to use the ideogram in a message sent by that user,or to view the ideogram in messages sent by other users. Userauthorization may be set by any of multiple methods, for example bymembership in a particular group, certain attributes of the user'sprofile, or by purchasing access to a sticker pack containing therestricted ideogram. If an unauthorized user attempts to use or view therestricted ideogram, the user may be presented instead with a warningmessage that he or she is unable to use or view the ideogram. Thewarning message may further inform the unauthorized user how he or shemay gain authorization.

In another particular embodiment, the author of a message may input textthat can be analyzed by the computing device for the sentiment containedwithin the text. Sentiment analysis conducted on at least a portion ofthe input text may result in the identification of one or moresentiments. One or more ideograms may then be suggested to the authorthat can substitute or supplement the analyzed text. The suggestion ofideograms may be user-specific, by the use of that individual author'sideogram usage history or personal ideogram dictionary. The suggestionmay also be based on the ideogram usage history of others connected tothe user, such as users sharing a common demographic attribute with theauthor or directed connected to the author in a social network.

In another particular embodiment, expression analysis may be performedon the subject(s) of a captured image. The expression analysis mayresult in one or more expressions displayed by the subjects of thecaptured image. One or more ideograms that can be substituted for thecaptured image may be suggested. The suggestion of ideograms may beuser-specific, by the use of that individual author's ideogram usagehistory or personal ideogram dictionary. The suggestion may also bebased on the ideogram usage history of others connected to the user,such as users sharing a common demographic attribute with the author ordirected connected to the author in a social network.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A illustrates a mobile device with a text message threadcontaining ideograms within the messages.

FIG. 1B illustrates a mobile device with a text message thread and atext entry area displayed, with the user typing in characters to triggerthe display of Unicode ideograms.

FIG. 1C illustrates a mobile device showing a text message thread and atext entry area, with a pull-down menu displaying a sticker pack ofideograms.

FIG. 2A illustrates a mobile device with a social network newsfeeddisplaying status update messages from other users that containideograms within the status updates.

FIG. 2B illustrates a mobile device displaying a status update userinterface, where the user may insert ideograms into a text statusupdate.

FIG. 2C illustrates a mobile device with a social network newsfeeddisplaying status update messages from other users that are filtered foronly updates containing a particular ideogram.

FIG. 2D illustrates a mobile device displaying a list of trendingideograms, sorted by popularity amongst direct connections in a socialnetwork.

FIG. 3 illustrates an example method of restricting access to use orviewing of particular ideograms that require user authorization.

FIG. 4A illustrates a mobile device showing an input text message wherethe author has attempted to insert a restricted ideogram and themessaging platform displays a warning to the author that the ideogram isrestricted.

FIG. 4B illustrates a mobile device that has sent a restricted ideogram,and an unauthorized recipient mobile device that displays a text messagewithout the restricted ideogram.

FIG. 5 illustrates an example method for suggesting ideograms to theauthor of a message based on sentiment analysis conducted on the textthat the author has input.

FIG. 6A illustrates a mobile device with a pop-up list of ideograms thatare suggested to the author based on sentiment analysis of thehighlighted portion of text.

FIG. 6B illustrates a mobile device with a social network newsfeeddisplaying status update messages from other users that includeideograms, with text descriptions of an ideogram appearing on thedisplay when the ideogram is tapped.

FIG. 6C illustrates a mobile device with a text message containing ahashtagged text phrase, and a recipient mobile device that hasincorporated the hashtagged text phrase into an ideogram.

FIG. 7 illustrates an example method for suggesting ideograms to theauthor of a message based on expression analysis conducted on a capturedimage including a person's face.

FIG. 8A illustrates a mobile device showing a front-facing camera viewof the user of the mobile device.

FIG. 8B illustrates a mobile device presenting the user with a list ofideograms based on the expression analysis conducted on the capturedimage.

FIG. 8C illustrates a mobile device presenting the user with a list ofideograms based on the expression analysis conducted on two subjects ina captured image.

FIG. 9 illustrates an example network environment associated with asocial-networking system.

FIG. 10 illustrates an example social graph.

FIG. 11 illustrates an example computer system.

DESCRIPTION OF EXAMPLE EMBODIMENTS

Particular embodiments feature the use of ideograms within visualdigital messages. Ideograms may be written characters, symbols or imagesthat represent an idea or thing without using letters or words; examplesmay be the gender symbols ♀ and ♂, or images of a bunny with a smilingor angry expression. Ideograms may be inserted into text using digitalimages presented to an author of the text in a “sticker pack”, where agroup of ideograms sharing a common theme or style may be displayed in auser interface enabling the author to select ideograms to be insertedinto the text. Alternatively ideograms may be inserted through standardslike Unicode which convert specific text strings to ideograms. Forexample, typing in the characters “:)” into an application thatrecognizes Unicode ideograms will convert those characters into theideogram

.

Messages may be sent between two computing devices using any of a numberof messaging platforms including but not limited to e-mail, ShortMessages Service (SMS), or instant message (IM). Messages may also be“broadcast” by an author to one or more recipients, e.g. through postinga message or status update on a social networking system, or by postinga message on a platform such as Twitter. An ideogram may be insertedinto a message and displayed within the image at any of a number ofpoints in time: they may be displayed when the author types in theideogram into the message; when the author chooses to send the finishedmessage; on the receiving device as the message is being received; orwhen the recipient opens the message to read the message.

FIG. 1A shows an example embodiment wherein ideograms depictingexpressive faces (sometimes called emoticons or “emoji”), may beembedded within the text 120A-E of a messaging platform 101 depicted ona front display 105 of a mobile device 100. The messaging platform 105may be configured to send and receive image files representing ideogramsor certain text strings corresponding to certain ideograms, and todisplay them to the author and recipient as an ideogram 121A within themessage field 120A.

FIG. 1B shows an example embodiment wherein the user authoring a message122 may insert one or more ideograms 121B into the body of the textmessage being sent. The messaging system may contain an auto-correctingsystem where the text message field automatically converts certain textstrings 125 into ideograms. This may occur through a standard such asUnicode. The author may select certain emoji to convey a messageconsistent with or supplementing the text content of the message 122.Ideograms may be placed at any position within the message, such as, byway of example and not limitation, the beginning, middle, or end of atext message. Alternatively, the author may choose to send a messagecontaining only one or more ideograms, without accompanying text.

FIG. 1C depicts an example embodiment where the author of a message maybe presented with a “sticker pack” window 130 of ideograms 135 to besent in the message. The sticker pack 130 may be displayed to the authorfor selection of one or more ideograms 135. The sticker pack 130 may bedisplayed based on user input, e.g. if the user inputs a hashtag symbol,#, or by selection of a dedicated button 137 on the messaging platform.A sticker pack 130 may consist of one or more ideograms 135 that hasbeen classified into a particular group 131. The author may select aparticular group from the category tabs 132A-C, and the sticker packwindow 130 displays the selected group of ideograms. For example,ideograms depicting a particular artistic theme or character (such asHappy Poo or Cute Bunny ideograms) may be grouped together. Othersticker packs may be selected by the author to insert other types ofideograms. The author's device 100 may be preloaded with certain sets ofsticker packs 130 for use in messaging; the author may also be presentedwith the option to purchase additional sticker packs containingadditional ideograms. Alternatively, the messaging system may beconfigured to present the author with the sticker pack of ideograms whencertain conditions are met, for example by displaying the sticker packsat the end of each sentence.

FIG. 2A depicts the use of ideograms in status messages in a socialnetwork news feed 210. A receiving device 200 receives status messagessent by the recipient's friends and acquaintances, and presents thestatus messages in a news feed to the recipient. The news feed containsone or more posts from other users, 220A-D. The status updates may alsocontain ideograms 225A-E contained within the body text of the statusupdates. The recipient may choose to comment on the status update, andmay insert ideograms into the body of the comments.

FIG. 2B depicts a user status update screen 230 for a social network,wherein the author may input a status or message 235 to be sent to oneor more other social network users. The group of users which may see thestatus update may be configured by the author. The status update inputfield may receive text, images, or video input by the author to post asa status update. The status update input field 235 may also beconfigured to present the author with the option of inserting ideograms245. In FIG. 2B, a particular embodiment may present the option ofinserting ideograms to the author in a sticker pack format 240.

FIG. 2C depicts an interface of a social network that allows a recipientto sort status updates in a news feed by the ideograms contained withineach status update. For example, the recipient may select any ideogramfrom the category tabs 250A-C. The ideograms displayed in the categorytab may be selected based upon actual use of the ideograms by others inthe social network; i.e. by displaying the ideograms in order of popularuse. Alternatively, the news feed may display only category tabs forideograms that the recipient has explicitly selected to be placed on thecategory tabs (e.g. if the recipient does not wish to see status updatessorted by a sad emoticon, they could remove that option from thecategory tabs 250A-C).

When the recipient selects a particular ideogram 225A from the categorytab 250A, the list of status updates in the news feed 210 may berefreshed to contain only messages containing the particular ideogramselected. For example, in FIG. 2C, the recipient has selected the tab250A containing an ideogram of a smiling face. The list of statusupdates below has filtered out messages that do not contain the sameideogram, and has listed only messages with the same ideogram.

The selection of a particular ideogram may be configured to showmessages or status updates that contain only that particular ideogram,or messages or status updates that can contain other status updates solong as the selected ideogram is also included. For example, statusupdates 220A and 220B both display the selected ideogram of a smilingface, but also contain other ideograms. A particular embodiment mayfilter those status updates out of the list as well, and only liststatus update 220D which contains only the selected ideogram. In anotherembodiment, the news feed may include status updates 220A and 220B inaddition to 220D.

FIG. 2D depicts an example embodiment where the recipient may view alist of ideograms 225 by a configured filter. The filter for display maybe the ideograms' popularity in use by other users of the socialnetwork. The list of ideograms may be sorted by their popularity forall-time, or over a certain subset of time, such as a particular monthor year. One embodiment may list the most popular ideograms over asubset of time dating to the present, such as the previous 7 or 30 days.This type of filtering, which may display the most popular ideogramsover the most recent period of time, may be called “trending”. Therecipient may view the list of ideograms 265A-E under the particularfilter 260, and view who if any friends or acquaintances have used thatparticular ideogram. Upon selection of one of the ideograms 265A-E inthe list, the recipient may be directed to a list of the actual statusupdates using each ideogram, similar to the interface in FIG. 2C.

FIG. 3 illustrates an example method 300 for providing ideograms into amessage wherein the ideograms may be restricted in use or in viewing bya recipient. An ideogram may be restricted only in use; that is, anunauthorized author may be prevented from inserting a restrictedideogram into a message. An ideogram may be additionally restricted inviewership, where if a recipient is not an authorized user of theideogram, he or she may be restricted from even viewing the ideogram.Determination of who is authorized to use or view a particular ideogramor set of ideograms may be based on conditions set by the author, or byanother entity. Restrictions to access to certain ideograms may includemembership in a certain group. Groups determining authorization mayconsist of, for example: members of a certain social club; people whohave purchased access rights to the particular ideogram or set ofideograms; direct connection to a particular user node or concept nodeof a social network; people with phone numbers with a certain area codeor prefix; or people who are registered users of a downloaded app ontheir mobile device. Restrictions to access may further be based onuser-specific information including a demographic attribute of the user(such as age, sex, gender, nationality, racial identification), aprofile attribute of the user on the messaging platform or socialnetworking system, a status of the user (e.g. the device is in “pairingmode” or “game mode”), parental controls set for the user's account, ora transaction history of the user. Transaction history requirements mayinclude requiring that users have played more than a preset number ofgames on a social-networking system, have attained a certain levelwithin a game or have played a game more than a certain number of hours,have posted a certain number of comments or status updates, or aredesignated as administrators or officers of an organization.

At step 310, ideograms may be selected for insertion into a message oras a standalone message. Examples of messages are a text message using amessaging platform to be sent to one or more recipients, or a postedstatus update of the author's profile on a social-networking system. Theexample method of FIG. 3 will embody a messaging platform.

At step 315, the messaging platform determines whether the ideogramselected by the author is restricted. If the author selects an ideogramwith no restrictions, the ideogram may be inserted into the message assoon as the author selects it, step 316. Upon selection of an ideogramwith restricted access, a request for authorization to access therestricted ideogram may be sent by the author's device.

At step 320, the messaging platform compares the author's user profileto a database of authorized users. An example method of this step may befor a messaging platform on a social-networking system to connect to aserver of the social-networking system to access a centralized database.Another example method may be for the authorization database to bestored locally on the author's computing device.

At step 325, the messaging platform may determine whether the author isauthorized to access the restricted ideogram. Determination ofauthorization may be based on whether the request for access comprisesan authorization code or token, e.g. a cookie or SSL certificate.

At step 326, if the author is not authorized, the messaging platform mayprevent the ideogram from being inserted, and may display a warningmessage informing the author that the ideogram is restricted. The authormay then choose to abandon insertion of the ideogram, or find a way togain authorization (e.g. by purchasing the sticker pack containing therestricted ideogram).

At step 330, if the author is an authorized user of the ideogram, therestricted ideogram may be displayed to the author. The messagingplatform may additionally determine if the author has determined therecipients who will view this message. The author may choose asrecipients one or several individuals, a specific group of individuals,or may be sending the message to a wider group, e.g. a status updatethat is viewable by any direct connections in a social network.

At step 335, if the author has selected one or more recipients, themessaging platform may determine if the recipients also are authorizedusers of the restricted ideogram. Determination of authorization for therecipients may be conducted similarly to determination of authorizationfor the author, as discussed in step 325.

At step 336, if one or more of the selected recipients are notauthorized, the messaging platform may display a warning to the authorthat the ideogram is restricted and that certain recipients may not beable to view the ideogram. The author may then have the option ofdeleting the ideogram from the message, or overriding the warning andproceeding with the message with the restricted ideogram.

At step 340, if all of the selected recipients are authorized, then themessaging platform may not display a warning and the ideogram will beinserted into the message. The ideogram may be placed at any number oflocations within the body of the message, for example at the beginningof the text, the end, or in the middle of the text. The text content ofthe message may wrap around the ideogram so that both the text andideogram are easily viewable.

At step 350, after the author has completed the message, he or she maychoose to send out the message to the intended one or more recipients.The messaging platform will send the message to the recipient devices,and the recipients will be able to view the message using a compatiblemessaging system.

At step 355, the messaging platforms on the recipient devices maydetermine again if the particular recipient is an authorized user of theideogram. This additional authorization check is used in case the authoroverrode the restricted-content response when the ideogram was insertedinto the message. Determination of authorization may be conducted in thesame fashion as described in steps 325 and 335.

At step 356, if the recipient is not authorized to view the ideogram,then the recipient may be able to view the message without therestricted ideogram; i.e. the ideogram will be deleted or redacted fromthe message. The messaging platform or social network interface mayinform the unauthorized recipient of the presence of a restrictedideogram. The recipient may then be presented with potential options forgaining authorization to that ideogram; e.g. by purchasing a stickerpack or by joining a certain group.

At step 360, if the recipients are authorized users of the restrictedideogram, they may view the message exactly as the author sent it, i.e.with the restricted ideograms.

Particular embodiments may repeat one or more steps of the method ofFIG. 3, where appropriate. Although this disclosure describes andillustrates particular steps of the method of FIG. 3 as occurring in aparticular order, this disclosure contemplates any suitable steps of themethod of FIG. 3 occurring in any suitable order. Moreover, althoughthis disclosure describes and illustrates particular components,devices, or systems carrying out particular steps of the method of FIG.3, this disclosure contemplates any suitable combination of any suitablecomponents, devices, or systems carrying out any suitable steps of themethod of FIG. 3.

FIG. 4A depicts restrictions being placed on the user of particularideograms. In one embodiment, the author may be attempting to insert anideogram 405 belonging to a particular sticker pack 132D into a textmessage 124. The ideogram 405 is restricted for use only by authorizedusers. Unauthorized users may be unable to use the particular ideogram405 or view the ideogram 405 in messages sent by authorized users. Thedevice may display a warning 410 to the author that the ideogram isrestricted. This warning may be sent for all uses of the restrictedideogram, or only if the messaging platform determines that the messageis being sent to an unauthorized recipient.

In FIG. 4B, the author has attempted to send a message 124 containing arestricted ideogram 405 to a device 200 belonging to a recipient who isnot an authorized user. While the author may be able to input therestricted ideogram 405 into the message 124, when the message 124 issent to the recipient, the ideogram 405 may have been deleted from thereceived message 120H. The recipient may see a message 120H containingthe rest of the sent message 124 without the restricted ideogram 405.

In particular embodiments, restrictions may serve to block the authorfrom sending particular ideograms or any ideograms at all to adesignated recipient. Restrictions may be based on any factors,including but not limited to: attributes of the author and/or therecipient (e.g., age or gender), the type of relationship between theauthor and the recipient (e.g., manager-employee or mother-son), time ofday, location. Such restrictions may be configured by the author, by therecipient, by an entity in control of licensing usage rights to theideogram, or by a site/system moderator.

FIG. 5 illustrates an example method 500 wherein the messaging platformmay suggest certain ideograms to the author of a message, based at leastpartly on the text content that the author has input. In the examplemethod of FIG. 5, the suggestions made by the messaging platform mayinclude sentiment analysis on the text content

At step 510, the author may input a text message into a messagingplatform. Some text messages may be devoid of any particular attachedsentiment, for example a reply of “yes.” Many other text messages maycontain one or more sentiments in the message.

At step 520, the messaging platform may select at least a part of theinput text message. As discussed in step 510 above, for some messagesthe messaging platform may determine that no part of the text can beselected for analysis. If the messaging platform does select at least aportion of the input message, the messaging platform may proceed to thenext step.

At step 530, the messaging platform may conduct sentiment analysis onthe selected text. Particular embodiments may incorporate anymethodology of performing sentiment analysis upon a word, phrase, orother body of text (which may or may not include ideograms), includingby way of example and not limitation: polarity classification, sentimentclassification according to a pre-defined set of emotional states,subjectivity/objectivity identification, or feature/aspect-basedsentiment analysis. Sentiment analysis may be based upon, not justwords, punctuation, and ideogram usage, but also other indicia, such as,by way of example and not limitation: (1) analysis of audio including avoice to detect volume, tone, and/or inflection, (2) analysis of videoto perform facial/gesture recognition and emotion detection, and/or (3)analysis of biometric sensor data to detect pulse, temperature, skinconductance, pressure and/or speed while typing/clicking on atouchscreen, and/or pupil constriction/dilation. At step 540, amessaging platform may identify one or more sentiments to be associatedwith the input text.

At step 550, the messaging platform may then compare the availableideograms to the identified one or more sentiments to suggest at leastone ideogram to the author of the input message. Factors incorporatedinto the sentiment analysis may include, by way of example and notlimitation: social networking information associated with the author ofthe message, information associated with the social networkingconnection between the author and one or more recipients of the message,membership of the author or recipient in a group authorized to use arestricted ideogram, an ideogram usage history for the author of themessage, or an ideogram usage history for other users of the socialnetwork system. In particular embodiments, methods for performingsentiment analysis may also include not just such information associatedwith the author or recipient(s) of a message, but also similarinformation associated with a “target” of a message: a person or conceptmentioned in the message. In one embodiment, the messaging platform mayhave indexed records in a database associating certain ideograms to asentiment. The indexed records may be user-specific; each author mayhave different indices for various sentiments and ideograms. Uponidentification of at least one sentiment, the messaging platform maydisplay the ideograms that are contained in the indexed record for thatsentiment, and present them to the author for selection.

Ideogram usage history for the author may be determined by consideringthe results of previous suggestions to the author; that is, if theauthor accepted a suggested ideogram in a similar situation, manuallyselected a different ideogram, or rejected the suggestion to include anideogram. Ideogram usage history may be further categorized based onadditional factors. For example, ideogram usage histories may be createdfor the author at different timeslots in the day (e.g. a history formessages sent between 10 pm and 3 am), for messages sent within acalendar range (e.g. a history of ideograms sent in the past year), formessages sent to particular recipients, for the current location of theauthor, or the current location of the identified recipient. Thesefactors and others may be used in sentiment analysis to generatedifferent suggestions for the author at different times, locations, orfor different recipients.

The ideogram usage history for other users in a social-networking systemmay be further categorized. Additional factors to create subsets ofusers may include users matching the author's demographic or locationprofile, users within a certain degree of separation from the author inthe social network graph, users who have previously or recentlycommunicated with the author, users who have previously or recentlycommunicated frequently with the author, or users associated to acertain group which the author has designated or to which the author isalso associated.

The indexed records of ideogram associations to a sentiment may becreated by the messaging platform provider, or a social network server.An ideogram may be associated with one or more sentiments, andconversely a sentiment may be associated with one or more ideograms. Thedetermination of associating an ideogram with a sentiment, as describedabove, may be quantified with a “score” of how well an ideogram isassociated with a sentiment based on the factors discussed above. Thescores may be defined as a division between two or more sentiments; e.g.a particular ideogram may be scored as 30% angry, 70% sad.

At step 560, the messaging platform may generate a list of ideograms tosuggest to the author at step 570, sorted by the score of each ideogramor by only displaying ideograms that are above a certain score. If thescores are a percentage split between two or more sentiments, themessaging platform may pick ideograms with the sentiment score within acertain tolerance threshold, e.g. by picking an ideogram with a scorewithin ±10% of the desired score for that sentiment. Furthermore, themessaging system may exclude ideograms that meet the predetermined scorerequirement, but is also associated with a second sentiment that themessaging platform or author has determined undesirable to be mixed withthe present sentiment.

At step 584, the scores may be updated through usage statisticscollected by the messaging platform. For example, if a particularideogram is selected a majority of the time for a particular sentimentby authors in a particular group or social network, then the messagingplatform may score that ideogram higher in its association with thesentiment, and present it first in the list of suggestions to futureauthors with the same sentiment. If the authors are manually picking anideogram not currently on the list of suggested ideograms, the messagingplatform may update the list to include the new ideogram. Conversely, ifan ideogram is not popular with authors for a particular sentiment, thatideogram may be placed farther down the list of suggestions, or droppedaltogether from the suggestions.

At step 582, ideograms may also be associated with metadata pertainingto that particular ideogram. Metadata to be associated with ideogramsmay include descriptive text for the ideogram, text input by the authorfor which the ideogram was substituted or added, or other data that isassociated with the author's use of that particular ideogram. Themessaging platform may create the indexed records for sentiments andideograms by including the metadata associated with each ideogram whengenerating the scores for each sentiment-ideogram association.

At step 580, the author will select one or more of the ideograms to beinserted into the message at step 590, substituting the analyzed text orbeing inserted into the text message without any deletions to the text.

Particular embodiments may repeat one or more steps of the method ofFIG. 5, where appropriate. Although this disclosure describes andillustrates particular steps of the method of FIG. 5 as occurring in aparticular order, this disclosure contemplates any suitable steps of themethod of FIG. 5 occurring in any suitable order. Moreover, althoughthis disclosure describes and illustrates particular components,devices, or systems carrying out particular steps of the method of FIG.5, this disclosure contemplates any suitable combination of any suitablecomponents, devices, or systems carrying out any suitable steps of themethod of FIG. 5.

FIG. 6A depicts an embodiment of sentiment analysis on a portion ofinput text 620 to suggest a set of ideograms 630 to the author.Ideograms may be arranged on an emotional range; for example, byarranging them in the following order: worried, anxious, frantic, angry,furious, upset, annoyed. Alternatively, ideograms may be arrangeddepending on their score or percentage score for the particularsentiment or emotion that is being represented. The messaging platformmay perform sentiment analysis on a part of the author's text 620. Theanalysis may then find the appropriate sentiments that the author istrying to express, based on methods described above. The messagingplatform may then display a pop-up menu to the author, highlight theanalyzed text, and display on the menu a list of ideograms thatcorrespond to the sentiment. The author may then choose an ideogram 635from the list, scroll further down the list, have the messaging platformconduct a new sentiment analysis, or cancel out of the pop-up menu. Ifthe author chooses to select and insert an ideogram from the displayedpop-up menu, the author may have the additional option of substitutingthe ideogram for the highlighted text 620, or adding the ideogram intothe unaltered text 610. Therefore, the ideogram selected by sentimentanalysis may either replace or supplement the text 620 upon which theanalysis was performed.

FIG. 6B depicts a mobile device 200 showing a newsfeed 210 of statusupdates 220A-D by users of a social network. The status updates mayincorporate ideograms within the text of the update, as seen above. Whena recipient viewing the update has any confusion about a particularideogram displayed, he or she may select the ideogram, for example witha tap 650 on the screen of the mobile device. The newsfeed may thendisplay to the recipient a text box 655 containing a text description ofthe ideogram.

The recipient or another user may interact with the displayed ideogramfurther. For example, the recipient may perform a double-tap,tap-and-hold, or other interaction with the ideogram, which may causethe messaging platform or social network interface to present optionssuch as viewing the popularity of the ideogram in the social network,refreshing a newsfeed to show status updates utilizing that ideogram, orinforming the recipient how he or she may gain authorization to use it.

FIG. 6C depicts an embodiment wherein the messaging platform selects anideogram 670 that incorporates a text input by the author. The authorwrites in a text message 126, and may designate a certain word or phrase660 to be incorporated into the ideogram. The word or phrase may bedesignated, for example, by prefacing it with a hashtag symbol, #, oranother character; the author may also type out the designated word orphrase, highlight or tap on the word or phrase, and push a button on themessaging platform that causes the platform to create an ideogramincorporating the word or phrase. In a particular embodiment, theideogram 670 may be a face accompanied by a thought or speech bubble675; the selected text 660 may then be displayed in a message 120I tothe recipient within the thought or speech bubble 675. The author may bepresented with options to alter the appearance of the ideogram; e.g. bychanging the expression on the ideogram face, or switching between athought bubble and a speech bubble.

FIG. 7 illustrates an example method 700 for suggesting ideograms to theauthor of a message based on an expression captured by a cameraassociated with the computing device. An expression may be defined as afacial expression associated with an emotion, a facial expression thathas a semantic association (which may not necessarily have a particularassociated emotion, such as a wink), a hand gesture (in combination witha facial expression or taken alone), or some other gesture that may bevisually captured.

At step 710, a camera application on the computing device may capture animage using an associated camera wherein one or more people areincluded. The associated camera may be integrated into the computingdevice, as shown in FIGS. 8A-8C (i.e., camera 110), or it may becommunicably connected to the computing device (e.g., a local webcamconnected by a USB cable, or a remote security camera connected via anetwork connection). FIG. 8A depicts an example embodiment of a mobiledevice 100 with a front-facing camera 110 adjacent to a front display105. In such a device 100, the camera application may utilize thefront-facing camera 110 to show the user a “self-portrait” view 805,where the user may easily see the image being captured on the frontdisplay of the mobile device. This captured image 810 may be a singleimage, a set of images (e.g., taken using a high-speed camera), or avideo clip or stream. The captured image may be stored on mobile device100, streamed from mobile device 100, uploaded to a server, or insertedinto a message to be sent to one or more recipients. In otherembodiments, the image may be captured while the camera displays a“live” feed on the display of mobile device 100 (i.e., without taking aphoto or shooting a video, or immediately prior to taking a photo orshooting a video) or utilized for data extraction and/or furtheranalysis (e.g., calculating facial geometry in order to perform facialrecognition or expression analysis).

At steps 720-728, the computing device or messaging platform may analyzethe captured image to classify one or more expressions displayed on theface of one or more people shown in the captured image. Analysis of thecaptured image may take place while the image is being captured, orafterwards (i.e., analysis of a previously-captured image). Theexpression analysis of the captured image may be conducted by anyapplication operable on the computing device, such as the messagingplatform, the camera application, a social-networking system, or anotherapplication. The application may additionally be associated with adatabase containing benchmark metrics for all available expressions, aswill be discussed below.

At step 722, the analysis of the faces displayed in the captured imagemay be conducted by first detecting the presence of one or more faces inthe captured image. Facial detection in captured images is a methodpreviously described in the art.

At step 724, the analysis application may then generate acharacterization of features of the face. For example, the applicationmay extract certain features or structures of the face by taking thedimensions or relative positions of certain features or structures. Theanalysis could also factor in skin texture analysis or athree-dimensional facial recognition analysis. The methods for facialrecognition where the various dimensions and characteristics of facialfeatures and structures are used to identify an individual arewell-known in the art. The analysis may be configured to only proceedwith the next step if the identified face is that of the author oranother person for whom the application has benchmark metrics (forexample, another user on a social-networking system).

The analysis of the face shown in the captured image may then generatequantified or qualified metrics or attributes of features, such as, forexample, shape, dimensions, texture, luminance, or color. For example,the analysis may generate metrics representing that the depicted eyesare at a certain angle and relative width, that the skin of the depictedface is pale or flushed, or that the depicted mouth is “upturned” or“downturned.”

At step 726, the generated metrics may then be compared to benchmarkmetrics stored in the database within the analysis application.Benchmark metrics may be defined as the numerical or comparative metricsthat correspond to a particular expression. The application may comparethe generated metrics for the captured image with the benchmark metricscorresponding to all the expressions stored by the application. Theapplication may then select one or more expressions that are associatedwith benchmark metrics that correspond closely to the generated metrics,and characterize the face in the captured image as having those one ormore expressions. Benchmark metrics may be associated with eachindividual user, where each identified user is associated with adifferent set of benchmark metrics. This means that two different userscould have very similar generated metrics, but have differentexpressions associated with the generated metrics due to the differencesin the benchmark metrics used.

The benchmark metrics may be generated by one of several methods. Forexample, the benchmark metrics may be user-specific. In an exampleembodiment, the user may be asked at a prior time to display aparticular expression or no expression, and the device may captureimages of the user for each displayed expression. The application mayanalyze these prior images to generate metrics for each image.

At step 728, the application may then use the metrics generated for eachimage as the benchmark metrics for the expression corresponding to thatimage. Benchmarks may also be created based on a group of subjects, forexample by a sampling of images of a population of users associated witha particular demographic profile, by a sampling of images of one or moreusers identified as family members of the user, or by a sampling ofimages of users associated with the user with respect to a socialnetwork.

At step 730, the ideograms that are usable by the messaging platform maybe associated with match scores that gives each ideogram a score for aparticular expression. An ideogram may be assigned match scores to oneor more expressions, and an expression may be associated with one ormore match scores, each corresponding to an ideogram. Once the messagingplatform has identified a particular expression displayed by a subjectin the analyzed photo, the platform may select one or more ideogramsthat are matched within a certain tolerance level, for example a matchscore above 60% of the maximum possible score.

The match scores associating an ideogram to a particular expression maybe calculated using a variety of factors. A generalized match scorecould be assigned using an ideogram dictionary, where the messagingplatform provider or another third party has assigned match scores toeach ideogram. The dictionary may have preset match scores for eachideogram that may be subsequently edited by the user. The messagingplatform may also take into consideration the ideogram usage history ofthe author. For example, if the author previously took an image ofhimself or herself that the expression analysis characterized as “sad”and the author selected a “pouty” ideogram, then the “pouty” ideogrammay have its match score increased for “sad”, and the next time theauthor takes a self-portrait that is characterized as “sad”, the “pouty”ideogram will be presented higher on the presented list of ideograms.Both the ideogram dictionary and ideogram usage history may beassociated only with the individual user, and may be updated only basedon the user's interactions with the ideogram match scores.Alternatively, the ideogram dictionary and usage history could beassociated with one or more social-networking connections of the user,associated with one or more users sharing common attributes with theuser (e.g. demographics, common interests, etc.), or based on generalstatistics regarding trending ideograms in a social-networking system.

At step 740, the messaging platform may then present the one or moreselected ideograms to the author of the message, for example in a listor a discrete sticker pack category alongside other sticker packcategories discussed above. FIG. 8B depicts the mobile device with afront-facing camera 110 and captured image 810 of the author. Themessaging platform using captured image 810 has conducted an expressionanalysis of the captured image, characterized an expression for theauthor's face in the captured image, and presented the author with alist of ideograms 820 that have high match scores to that expression.The list or sticker pack of ideograms may be arranged with the ideograms825 having the highest match scores first, down to the lowest-scoringideograms that still exceed the tolerance threshold. In someembodiments, the above-described steps (e.g., image capture, expressionanalysis, and ideogram matching) may be performed prior to taking aphoto or video.

FIG. 8C depicts a particular embodiment wherein captured image 830selected has more than one subject 832 and 834. The mobile device 100 ormessaging platform 101 may recognize that captured image 830 has one ormore subjects, and accordingly present a list of ideograms 840 to theauthor that depict one or more expressive ideograms 841. In the exampleembodiment of FIG. 8C, the messaging platform may conduct expressionanalysis and recognize that the foreground subject 832 of captured image830 has a sad or somber expression, while the background subject 834 hasan amused or happy expression. Accordingly, the ideograms that arerecommended for selection by the author may have two faces, with theforeground ideogram 841A having a sad expression and the backgroundideogram 841B having a happier expression.

At step 750, upon presentation of the one or more suggested ideograms,the author may select one or more of the ideograms, and at step 760,insert the selected one or more ideograms into a message to be sent toone or more recipients. Alternatively, the author may be presented witha user interface to post the selected ideograms for annotation ofanother content item on a messaging platform or social network. Acontent item may include text, image, video, audio, or other contentposted by the user, another user, or a third party. For example, ifanother user has posted a photograph on a social network, instead of“liking” the photograph, the author may annotate the photograph with aselected ideogram that is presented to the author using expressionanalysis. The author may take a self-portrait of himself or herselfreacting to the posted photograph, and then select one or more ideogramsthat are presented to the author after expression analysis.

Particular embodiments may repeat one or more steps of the method ofFIG. 7, where appropriate. Although this disclosure describes andillustrates particular steps of the method of FIG. 7 as occurring in aparticular order, this disclosure contemplates any suitable steps of themethod of FIG. 7 occurring in any suitable order. Moreover, althoughthis disclosure describes and illustrates particular components,devices, or systems carrying out particular steps of the method of FIG.7, this disclosure contemplates any suitable combination of any suitablecomponents, devices, or systems carrying out any suitable steps of themethod of FIG. 7.

FIG. 9 illustrates an example network environment 900 associated with asocial-networking system. Network environment 900 includes a clientsystem 930, a social-networking system 960, and a third-party system 970connected to each other by a network 910. Although FIG. 9 illustrates aparticular arrangement of client system 930, social-networking system960, third-party system 970, and network 910, this disclosurecontemplates any suitable arrangement of client system 930,social-networking system 960, third-party system 970, and network 910.As an example and not by way of limitation, two or more of client system930, social-networking system 960, and third-party system 970 may beconnected to each other directly, bypassing network 910. As anotherexample, two or more of client system 930, social-networking system 960,and third-party system 970 may be physically or logically co-locatedwith each other in whole or in part. Moreover, although FIG. 9illustrates a particular number of client systems 930, social-networkingsystems 960, third-party systems 970, and networks 910, this disclosurecontemplates any suitable number of client systems 930,social-networking systems 960, third-party systems 970, and networks910. As an example and not by way of limitation, network environment 900may include multiple client system 930, social-networking systems 960,third-party systems 970, and networks 910.

This disclosure contemplates any suitable network 910. As an example andnot by way of limitation, one or more portions of network 910 mayinclude an ad hoc network, an intranet, an extranet, a virtual privatenetwork (VPN), a local area network (LAN), a wireless LAN (WLAN), a widearea network (WAN), a wireless WAN (WWAN), a metropolitan area network(MAN), a portion of the Internet, a portion of the Public SwitchedTelephone Network (PSTN), a cellular telephone network, or a combinationof two or more of these. Network 910 may include one or more networks910.

Links 950 may connect client system 930, social-networking system 960,and third-party system 970 to communication network 910 or to eachother. This disclosure contemplates any suitable links 950. Inparticular embodiments, one or more links 950 include one or morewireline (such as for example Digital Subscriber Line (DSL) or Data OverCable Service Interface Specification (DOCSIS)), wireless (such as forexample Wi-Fi or Worldwide Interoperability for Microwave Access(WiMAX)), or optical (such as for example Synchronous Optical Network(SONET) or Synchronous Digital Hierarchy (SDH)) links. In particularembodiments, one or more links 950 each include an ad hoc network, anintranet, an extranet, a VPN, a LAN, a WLAN, a WAN, a WWAN, a MAN, aportion of the Internet, a portion of the PSTN, a cellulartechnology-based network, a satellite communications technology-basednetwork, another link 950, or a combination of two or more such links950. Links 950 need not necessarily be the same throughout networkenvironment 900. One or more first links 950 may differ in one or morerespects from one or more second links 950.

In particular embodiments, client system 930 may be an electronic deviceincluding hardware, software, or embedded logic components or acombination of two or more such components and capable of carrying outthe appropriate functionalities implemented or supported by clientsystem 930. As an example and not by way of limitation, a client system930 may include a computer system such as a desktop computer, notebookor laptop computer, netbook, a tablet computer, e-book reader, GPSdevice, camera, personal digital assistant (PDA), handheld electronicdevice, cellular telephone, smartphone, other suitable electronicdevice, or any suitable combination thereof. This disclosurecontemplates any suitable client systems 930. A client system 930 mayenable a network user at client system 930 to access network 910. Aclient system 930 may enable its user to communicate with other users atother client systems 930.

In particular embodiments, client system 930 may include a web browser932, such as MICROSOFT INTERNET EXPLORER, GOOGLE CHROME or MOZILLAFIREFOX, and may have one or more add-ons, plug-ins, or otherextensions, such as TOOLBAR or YAHOO TOOLBAR. A user at client system930 may enter a Uniform Resource Locator (URL) or other addressdirecting the web browser 932 to a particular server (such as server962, or a server associated with a third-party system 970), and the webbrowser 932 may generate a Hyper Text Transfer Protocol (HTTP) requestand communicate the HTTP request to server. The server may accept theHTTP request and communicate to client system 930 one or more Hyper TextMarkup Language (HTML) files responsive to the HTTP request. Clientsystem 930 may render a webpage based on the HTML files from the serverfor presentation to the user. This disclosure contemplates any suitablewebpage files. As an example and not by way of limitation, webpages mayrender from HTML files, Extensible Hyper Text Markup Language (XHTML)files, or Extensible Markup Language (XML) files, according toparticular needs. Such pages may also execute scripts such as, forexample and without limitation, those written in JAVASCRIPT, JAVA,MICROSOFT SILVERLIGHT, combinations of markup language and scripts suchas AJAX (Asynchronous JAVASCRIPT and XML), and the like. Herein,reference to a webpage encompasses one or more corresponding webpagefiles (which a browser may use to render the webpage) and vice versa,where appropriate.

In particular embodiments, social-networking system 960 may be anetwork-addressable computing system that can host an online socialnetwork. Social-networking system 960 may generate, store, receive, andsend social-networking data, such as, for example, user-profile data,concept-profile data, social-graph information, or other suitable datarelated to the online social network. Social-networking system 960 maybe accessed by the other components of network environment 900 eitherdirectly or via network 910. In particular embodiments,social-networking system 960 may include one or more servers 962. Eachserver 962 may be a unitary server or a distributed server spanningmultiple computers or multiple datacenters. Servers 962 may be ofvarious types, such as, for example and without limitation, web server,news server, mail server, message server, advertising server, fileserver, application server, exchange server, database server, proxyserver, another server suitable for performing functions or processesdescribed herein, or any combination thereof. In particular embodiments,each server 962 may include hardware, software, or embedded logiccomponents or a combination of two or more such components for carryingout the appropriate functionalities implemented or supported by server962. In particular embodiments, social-networking system 964 may includeone or more data stores 964. Data stores 964 may be used to storevarious types of information. In particular embodiments, the informationstored in data stores 964 may be organized according to specific datastructures. In particular embodiments, each data store 964 may be arelational, columnar, correlation, or other suitable database. Althoughthis disclosure describes or illustrates particular types of databases,this disclosure contemplates any suitable types of databases. Particularembodiments may provide interfaces that enable a client system 930, asocial-networking system 960, or a third-party system 970 to manage,retrieve, modify, add, or delete, the information stored in data store964.

In particular embodiments, social-networking system 960 may store one ormore social graphs in one or more data stores 964. In particularembodiments, a social graph may include multiple nodes—which may includemultiple user nodes (each corresponding to a particular user) ormultiple concept nodes (each corresponding to a particular concept)—andmultiple edges connecting the nodes. Social-networking system 960 mayprovide users of the online social network the ability to communicateand interact with other users. In particular embodiments, users may jointhe online social network via social-networking system 960 and then addconnections (e.g., relationships) to a number of other users ofsocial-networking system 960 whom they want to be connected to. Herein,the term “friend” may refer to any other user of social-networkingsystem 960 with whom a user has formed a connection, association, orrelationship via social-networking system 960.

In particular embodiments, social-networking system 960 may provideusers with the ability to take actions on various types of items orobjects, supported by social-networking system 960. As an example andnot by way of limitation, the items and objects may include groups orsocial networks to which users of social-networking system 960 maybelong, events or calendar entries in which a user might be interested,computer-based applications that a user may use, transactions that allowusers to buy or sell items via the service, interactions withadvertisements that a user may perform, or other suitable items orobjects. A user may interact with anything that is capable of beingrepresented in social-networking system 960 or by an external system ofthird-party system 970, which is separate from social-networking system960 and coupled to social-networking system 960 via a network 910.

In particular embodiments, social-networking system 960 may be capableof linking a variety of entities. As an example and not by way oflimitation, social-networking system 960 may enable users to interactwith each other as well as receive content from third-party systems 970or other entities, or to allow users to interact with these entitiesthrough an application programming interfaces (API) or othercommunication channels.

In particular embodiments, a third-party system 970 may include one ormore types of servers, one or more data stores, one or more interfaces,including but not limited to APIs, one or more web services, one or morecontent sources, one or more networks, or any other suitable components,e.g., that servers may communicate with. A third-party system 970 may beoperated by a different entity from an entity operatingsocial-networking system 960. In particular embodiments, however,social-networking system 960 and third-party systems 970 may operate inconjunction with each other to provide social-networking services tousers of social-networking system 960 or third-party systems 970. Inthis sense, social-networking system 960 may provide a platform, orbackbone, which other systems, such as third-party systems 970, may useto provide social-networking services and functionality to users acrossthe Internet.

In particular embodiments, a third-party system 970 may include athird-party content object provider. A third-party content objectprovider may include one or more sources of content objects, which maybe communicated to a client system 930. As an example and not by way oflimitation, content objects may include information regarding things oractivities of interest to the user, such as, for example, movie showtimes, movie reviews, restaurant reviews, restaurant menus, productinformation and reviews, or other suitable information. As anotherexample and not by way of limitation, content objects may includeincentive content objects, such as coupons, discount tickets, giftcertificates, or other suitable incentive objects.

In particular embodiments, social-networking system 960 also includesuser-generated content objects, which may enhance a user's interactionswith social-networking system 960. User-generated content may includeanything a user can add, upload, send, or “post” to social-networkingsystem 960. As an example and not by way of limitation, a usercommunicates posts to social-networking system 960 from a client system930. Posts may include data such as status updates or other textualdata, location information, photos, videos, links, music or othersimilar data or media. Content may also be added to social-networkingsystem 960 by a third-party through a “communication channel,” such as anewsfeed or stream.

In particular embodiments, social-networking system 960 may include avariety of servers, sub-systems, programs, modules, logs, and datastores. In particular embodiments, social-networking system 960 mayinclude one or more of the following: a web server, action logger,API-request server, relevance-and-ranking engine, content-objectclassifier, notification controller, action log,third-party-content-object-exposure log, inference module,authorization/privacy server, search module, advertisement-targetingmodule, user-interface module, user-profile store, connection store,third-party content store, or location store. Social-networking system960 may also include suitable components such as network interfaces,security mechanisms, load balancers, failover servers,management-and-network-operations consoles, other suitable components,or any suitable combination thereof. In particular embodiments,social-networking system 960 may include one or more user-profile storesfor storing user profiles. A user profile may include, for example,biographic information, demographic information, behavioral information,social information, or other types of descriptive information, such aswork experience, educational history, hobbies or preferences, interests,affinities, or location. Interest information may include interestsrelated to one or more categories. Categories may be general orspecific. As an example and not by way of limitation, if a user “likes”an article about a brand of shoes the category may be the brand, or thegeneral category of “shoes” or “clothing.” A connection store may beused for storing connection information about users. The connectioninformation may indicate users who have similar or common workexperience, group memberships, hobbies, educational history, or are inany way related or share common attributes. The connection informationmay also include user-defined connections between different users andcontent (both internal and external). A web server may be used forlinking social-networking system 960 to one or more client systems 930or one or more third-party system 970 via network 910. The web servermay include a mail server or other messaging functionality for receivingand routing messages between social-networking system 960 and one ormore client systems 930. An API-request server may allow a third-partysystem 970 to access information from social-networking system 960 bycalling one or more APIs. An action logger may be used to receivecommunications from a web server about a user's actions on or offsocial-networking system 960. In conjunction with the action log, athird-party-content-object log may be maintained of user exposures tothird-party-content objects. A notification controller may provideinformation regarding content objects to a client system 930.Information may be pushed to a client system 930 as notifications, orinformation may be pulled from client system 930 responsive to a requestreceived from client system 930. Authorization servers may be used toenforce one or more privacy settings of the users of social-networkingsystem 960. A privacy setting of a user determines how particularinformation associated with a user can be shared. The authorizationserver may allow users to opt in to or opt out of having their actionslogged by social-networking system 960 or shared with other systems(e.g., third-party system 970), such as, for example, by settingappropriate privacy settings. Third-party-content-object stores may beused to store content objects received from third parties, such as athird-party system 970. Location stores may be used for storing locationinformation received from client systems 930 associated with users.Advertisement-pricing modules may combine social information, thecurrent time, location information, or other suitable information toprovide relevant advertisements, in the form of notifications, to auser.

FIG. 10 illustrates example social graph 1000. In particularembodiments, social-networking system 960 may store one or more socialgraphs 1000 in one or more data stores. In particular embodiments,social graph 1000 may include multiple nodes—which may include multipleuser nodes 1002 or multiple concept nodes 1004—and multiple edges 1006connecting the nodes. Example social graph 1000 illustrated in FIG. 10is shown, for didactic purposes, in a two-dimensional visual maprepresentation. In particular embodiments, a social-networking system960, client system 930, or third-party system 970 may access socialgraph 1000 and related social-graph information for suitableapplications. The nodes and edges of social graph 1000 may be stored asdata objects, for example, in a data store (such as a social-graphdatabase). Such a data store may include one or more searchable orqueryable indexes of nodes or edges of social graph 1000.

In particular embodiments, a user node 1002 may correspond to a user ofsocial-networking system 960. As an example and not by way oflimitation, a user may be an individual (human user), an entity (e.g.,an enterprise, business, or third-party application), or a group (e.g.,of individuals or entities) that interacts or communicates with or oversocial-networking system 960. In particular embodiments, when a userregisters for an account with social-networking system 960,social-networking system 960 may create a user node 1002 correspondingto the user, and store the user node 1002 in one or more data stores.Users and user nodes 1002 described herein may, where appropriate, referto registered users and user nodes 1002 associated with registeredusers. In addition or as an alternative, users and user nodes 1002described herein may, where appropriate, refer to users that have notregistered with social-networking system 960. In particular embodiments,a user node 1002 may be associated with information provided by a useror information gathered by various systems, including social-networkingsystem 960. As an example and not by way of limitation, a user mayprovide his or her name, profile picture, contact information, birthdate, sex, marital status, family status, employment, educationbackground, preferences, interests, or other demographic information. Inparticular embodiments, a user node 1002 may be associated with one ormore data objects corresponding to information associated with a user.In particular embodiments, a user node 1002 may correspond to one ormore webpages.

In particular embodiments, a concept node 1004 may correspond to aconcept. As an example and not by way of limitation, a concept maycorrespond to a place (such as, for example, a movie theater,restaurant, landmark, or city); a website (such as, for example, awebsite associated with social-network system 960 or a third-partywebsite associated with a web-application server); an entity (such as,for example, a person, business, group, sports team, or celebrity); aresource (such as, for example, an audio file, video file, digitalphoto, text file, structured document, or application) which may belocated within social-networking system 960 or on an external server,such as a web-application server; real or intellectual property (suchas, for example, a sculpture, painting, movie, game, song, idea,photograph, or written work); a game; an activity; an idea or theory;another suitable concept; or two or more such concepts. A concept node1004 may be associated with information of a concept provided by a useror information gathered by various systems, including social-networkingsystem 960. As an example and not by way of limitation, information of aconcept may include a name or a title; one or more images (e.g., animage of the cover page of a book); a location (e.g., an address or ageographical location); a website (which may be associated with a URL);contact information (e.g., a phone number or an email address); othersuitable concept information; or any suitable combination of suchinformation. In particular embodiments, a concept node 1004 may beassociated with one or more data objects corresponding to informationassociated with concept node 1004. In particular embodiments, a conceptnode 1004 may correspond to one or more webpages.

In particular embodiments, a node in social graph 1000 may represent orbe represented by a webpage (which may be referred to as a “profilepage”). Profile pages may be hosted by or accessible tosocial-networking system 960. Profile pages may also be hosted onthird-party websites associated with a third-party server 970. As anexample and not by way of limitation, a profile page corresponding to aparticular external webpage may be the particular external webpage andthe profile page may correspond to a particular concept node 1004.Profile pages may be viewable by all or a selected subset of otherusers. As an example and not by way of limitation, a user node 1002 mayhave a corresponding user-profile page in which the corresponding usermay add content, make declarations, or otherwise express himself orherself. As another example and not by way of limitation, a concept node1004 may have a corresponding concept-profile page in which one or moreusers may add content, make declarations, or express themselves,particularly in relation to the concept corresponding to concept node1004.

In particular embodiments, a concept node 1004 may represent athird-party webpage or resource hosted by a third-party system 970. Thethird-party webpage or resource may include, among other elements,content, a selectable or other icon, or other inter-actable object(which may be implemented, for example, in JavaScript, AJAX, or PHPcodes) representing an action or activity. As an example and not by wayof limitation, a third-party webpage may include a selectable icon suchas “like,” “check in,” “eat,” “recommend,” or another suitable action oractivity. A user viewing the third-party webpage may perform an actionby selecting one of the icons (e.g., “eat”), causing a client system 930to send to social-networking system 960 a message indicating the user'saction. In response to the message, social-networking system 960 maycreate an edge (e.g., an “eat” edge) between a user node 1002corresponding to the user and a concept node 1004 corresponding to thethird-party webpage or resource and store edge 1006 in one or more datastores.

In particular embodiments, a pair of nodes in social graph 1000 may beconnected to each other by one or more edges 1006. An edge 1006connecting a pair of nodes may represent a relationship between the pairof nodes. In particular embodiments, an edge 1006 may include orrepresent one or more data objects or attributes corresponding to therelationship between a pair of nodes. As an example and not by way oflimitation, a first user may indicate that a second user is a “friend”of the first user. In response to this indication, social-networkingsystem 960 may send a “friend request” to the second user. If the seconduser confirms the “friend request,” social-networking system 960 maycreate an edge 1006 connecting the first user's user node 1002 to thesecond user's user node 1002 in social graph 1000 and store edge 1006 associal-graph information in one or more of data stores 964. In theexample of FIG. 10, social graph 1000 includes an edge 1006 indicating afriend relation between user nodes 1002 of user “A” and user “B” and anedge indicating a friend relation between user nodes 1002 of user “C”and user “B.” Although this disclosure describes or illustratesparticular edges 1006 with particular attributes connecting particularuser nodes 1002, this disclosure contemplates any suitable edges 1006with any suitable attributes connecting user nodes 1002. As an exampleand not by way of limitation, an edge 1006 may represent a friendship,family relationship, business or employment relationship, fanrelationship, follower relationship, visitor relationship, subscriberrelationship, superior/subordinate relationship, reciprocalrelationship, non-reciprocal relationship, another suitable type ofrelationship, or two or more such relationships. Moreover, although thisdisclosure generally describes nodes as being connected, this disclosurealso describes users or concepts as being connected. Herein, referencesto users or concepts being connected may, where appropriate, refer tothe nodes corresponding to those users or concepts being connected insocial graph 1000 by one or more edges 1006.

In particular embodiments, an edge 1006 between a user node 1002 and aconcept node 1004 may represent a particular action or activityperformed by a user associated with user node 1002 toward a conceptassociated with a concept node 1004. As an example and not by way oflimitation, as illustrated in FIG. 10, a user may “like,” “attended,”“played,” “listened,” “cooked,” “worked at,” or “watched” a concept,each of which may correspond to a edge type or subtype. Aconcept-profile page corresponding to a concept node 1004 may include,for example, a selectable “check in” icon (such as, for example, aclickable “check in” icon) or a selectable “add to favorites” icon.Similarly, after a user clicks these icons, social-networking system 960may create a “favorite” edge or a “check in” edge in response to auser's action corresponding to a respective action. As another exampleand not by way of limitation, a user (user “C”) may listen to aparticular song (“Ramble On”) using a particular application (SPOTIFY,which is an online music application). In this case, social-networkingsystem 960 may create a “listened” edge 1006 and a “used” edge (asillustrated in FIG. 10) between user nodes 1002 corresponding to theuser and concept nodes 1004 corresponding to the song and application toindicate that the user listened to the song and used the application.Moreover, social-networking system 960 may create a “played” edge 1006(as illustrated in FIG. 10) between concept nodes 1004 corresponding tothe song and the application to indicate that the particular song wasplayed by the particular application. In this case, “played” edge 1006corresponds to an action performed by an external application (SPOTIFY)on an external audio file (the song “Imagine”). Although this disclosuredescribes particular edges 1006 with particular attributes connectinguser nodes 1002 and concept nodes 1004, this disclosure contemplates anysuitable edges 1006 with any suitable attributes connecting user nodes1002 and concept nodes 1004. Moreover, although this disclosuredescribes edges between a user node 1002 and a concept node 1004representing a single relationship, this disclosure contemplates edgesbetween a user node 1002 and a concept node 1004 representing one ormore relationships. As an example and not by way of limitation, an edge1006 may represent both that a user likes and has used at a particularconcept. Alternatively, another edge 1006 may represent each type ofrelationship (or multiples of a single relationship) between a user node1002 and a concept node 1004 (as illustrated in FIG. 10 between usernode 1002 for user “E” and concept node 1004 for “SPOTIFY”).

In particular embodiments, social-networking system 960 may create anedge 1006 between a user node 1002 and a concept node 1004 in socialgraph 1000. As an example and not by way of limitation, a user viewing aconcept-profile page (such as, for example, by using a web browser or aspecial-purpose application hosted by the user's client system 930) mayindicate that he or she likes the concept represented by the conceptnode 1004 by clicking or selecting a “Like” icon, which may cause theuser's client system 930 to send to social-networking system 960 amessage indicating the user's liking of the concept associated with theconcept-profile page. In response to the message, social-networkingsystem 960 may create an edge 1006 between user node 1002 associatedwith the user and concept node 1004, as illustrated by “like” edge 1006between the user and concept node 1004. In particular embodiments,social-networking system 960 may store an edge 1006 in one or more datastores. In particular embodiments, an edge 1006 may be automaticallyformed by social-networking system 960 in response to a particular useraction. As an example and not by way of limitation, if a first useruploads a picture, watches a movie, or listens to a song, an edge 1006may be formed between user node 1002 corresponding to the first user andconcept nodes 1004 corresponding to those concepts. Although thisdisclosure describes forming particular edges 1006 in particularmanners, this disclosure contemplates forming any suitable edges 1006 inany suitable manner.

In particular embodiments, an advertisement may be text (which may beHTML-linked), one or more images (which may be HTML-linked), one or morevideos, audio, one or more ADOBE FLASH files, a suitable combination ofthese, or any other suitable advertisement in any suitable digitalformat presented on one or more webpages, in one or more e-mails, or inconnection with search results requested by a user. In addition or as analternative, an advertisement may be one or more sponsored stories(e.g., a news-feed or ticker item on social-networking system 960). Asponsored story may be a social action by a user (such as “liking” apage, “liking” or commenting on a post on a page, RSVPing to an eventassociated with a page, voting on a question posted on a page, checkingin to a place, using an application or playing a game, or “liking” orsharing a website) that an advertiser promotes, for example, by havingthe social action presented within a pre-determined area of a profilepage of a user or other page, presented with additional informationassociated with the advertiser, bumped up or otherwise highlightedwithin news feeds or tickers of other users, or otherwise promoted. Theadvertiser may pay to have the social action promoted. As an example andnot by way of limitation, advertisements may be included among thesearch results of a search-results page, where sponsored content ispromoted over non-sponsored content.

In particular embodiments, an advertisement may be requested for displaywithin social-networking-system webpages, third-party webpages, or otherpages. An advertisement may be displayed in a dedicated portion of apage, such as in a banner area at the top of the page, in a column atthe side of the page, in a GUI of the page, in a pop-up window, in adrop-down menu, in an input field of the page, over the top of contentof the page, or elsewhere with respect to the page. In addition or as analternative, an advertisement may be displayed within an application. Anadvertisement may be displayed within dedicated pages, requiring theuser to interact with or watch the advertisement before the user mayaccess a page or utilize an application. The user may, for example viewthe advertisement through a web browser.

A user may interact with an advertisement in any suitable manner. Theuser may click or otherwise select the advertisement. By selecting theadvertisement, the user may be directed to (or a browser or otherapplication being used by the user) a page associated with theadvertisement. At the page associated with the advertisement, the usermay take additional actions, such as purchasing a product or serviceassociated with the advertisement, receiving information associated withthe advertisement, or subscribing to a newsletter associated with theadvertisement. An advertisement with audio or video may be played byselecting a component of the advertisement (like a “play button”).Alternatively, by selecting the advertisement, social-networking system960 may execute or modify a particular action of the user.

An advertisement may also include social-networking-system functionalitythat a user may interact with. As an example and not by way oflimitation, an advertisement may enable a user to “like” or otherwiseendorse the advertisement by selecting an icon or link associated withendorsement. As another example and not by way of limitation, anadvertisement may enable a user to search (e.g., by executing a query)for content related to the advertiser. Similarly, a user may share theadvertisement with another user (e.g., through social-networking system960) or RSVP (e.g., through social-networking system 960) to an eventassociated with the advertisement. In addition or as an alternative, anadvertisement may include social-networking-system context directed tothe user. As an example and not by way of limitation, an advertisementmay display information about a friend of the user withinsocial-networking system 960 who has taken an action associated with thesubject matter of the advertisement.

In particular embodiments, social-networking system 960 may determinethe social-graph affinity (which may be referred to herein as“affinity”) of various social-graph entities for each other. Affinitymay represent the strength of a relationship or level of interestbetween particular objects associated with the online social network,such as users, concepts, content, actions, advertisements, other objectsassociated with the online social network, or any suitable combinationthereof. Affinity may also be determined with respect to objectsassociated with third-party systems 970 or other suitable systems. Anoverall affinity for a social-graph entity for each user, subjectmatter, or type of content may be established. The overall affinity maychange based on continued monitoring of the actions or relationshipsassociated with the social-graph entity. Although this disclosuredescribes determining particular affinities in a particular manner, thisdisclosure contemplates determining any suitable affinities in anysuitable manner.

In particular embodiments, social-networking system 960 may measure orquantify social-graph affinity using an affinity coefficient (which maybe referred to herein as “coefficient”). The coefficient may representor quantify the strength of a relationship between particular objectsassociated with the online social network. The coefficient may alsorepresent a probability or function that measures a predictedprobability that a user will perform a particular action based on theuser's interest in the action. In this way, a user's future actions maybe predicted based on the user's prior actions, where the coefficientmay be calculated at least in part a the history of the user's actions.Coefficients may be used to predict any number of actions, which may bewithin or outside of the online social network. As an example and not byway of limitation, these actions may include various types ofcommunications, such as sending messages, posting content, or commentingon content; various types of a observation actions, such as accessing orviewing profile pages, media, or other suitable content; various typesof coincidence information about two or more social-graph entities, suchas being in the same group, tagged in the same photograph, checked-in atthe same location, or attending the same event; or other suitableactions. Although this disclosure describes measuring affinity in aparticular manner, this disclosure contemplates measuring affinity inany suitable manner.

In particular embodiments, social-networking system 960 may use avariety of factors to calculate a coefficient. These factors mayinclude, for example, user actions, types of relationships betweenobjects, location information, other suitable factors, or anycombination thereof. In particular embodiments, different factors may beweighted differently when calculating the coefficient. The weights foreach factor may be static or the weights may change according to, forexample, the user, the type of relationship, the type of action, theuser's location, and so forth. Ratings for the factors may be combinedaccording to their weights to determine an overall coefficient for theuser. As an example and not by way of limitation, particular useractions may be assigned both a rating and a weight while a relationshipassociated with the particular user action is assigned a rating and acorrelating weight (e.g., so the weights total 100%). To calculate thecoefficient of a user towards a particular object, the rating assignedto the user's actions may comprise, for example, 60% of the overallcoefficient, while the relationship between the user and the object maycomprise 40% of the overall coefficient. In particular embodiments, thesocial-networking system 960 may consider a variety of variables whendetermining weights for various factors used to calculate a coefficient,such as, for example, the time since information was accessed, decayfactors, frequency of access, relationship to information orrelationship to the object about which information was accessed,relationship to social-graph entities connected to the object, short- orlong-term averages of user actions, user feedback, other suitablevariables, or any combination thereof. As an example and not by way oflimitation, a coefficient may include a decay factor that causes thestrength of the signal provided by particular actions to decay withtime, such that more recent actions are more relevant when calculatingthe coefficient. The ratings and weights may be continuously updatedbased on continued tracking of the actions upon which the coefficient isbased. Any type of process or algorithm may be employed for assigning,combining, averaging, and so forth the ratings for each factor and theweights assigned to the factors. In particular embodiments,social-networking system 960 may determine coefficients usingmachine-learning algorithms trained on historical actions and past userresponses, or data farmed from users by exposing them to various optionsand measuring responses. Although this disclosure describes calculatingcoefficients in a particular manner, this disclosure contemplatescalculating coefficients in any suitable manner.

In particular embodiments, social-networking system 960 may calculate acoefficient based on a user's actions. Social-networking system 960 maymonitor such actions on the online social network, on a third-partysystem 970, on other suitable systems, or any combination thereof. Anysuitable type of user actions may be tracked or monitored. Typical useractions include viewing profile pages, creating or posting content,interacting with content, joining groups, listing and confirmingattendance at events, checking-in at locations, liking particular pages,creating pages, and performing other tasks that facilitate socialaction. In particular embodiments, social-networking system 960 maycalculate a coefficient based on the user's actions with particulartypes of content. The content may be associated with the online socialnetwork, a third-party system 970, or another suitable system. Thecontent may include users, profile pages, posts, news stories,headlines, instant messages, chat room conversations, emails,advertisements, pictures, video, music, other suitable objects, or anycombination thereof. Social-networking system 960 may analyze a user'sactions to determine whether one or more of the actions indicate anaffinity for subject matter, content, other users, and so forth. As anexample and not by way of limitation, if a user may make frequentlyposts content related to “coffee” or variants thereof, social-networkingsystem 960 may determine the user has a high coefficient with respect tothe concept “coffee”. Particular actions or types of actions may beassigned a higher weight and/or rating than other actions, which mayaffect the overall calculated coefficient. As an example and not by wayof limitation, if a first user emails a second user, the weight or therating for the action may be higher than if the first user simply viewsthe user-profile page for the second user.

In particular embodiments, social-networking system 960 may calculate acoefficient based on the type of relationship between particularobjects. Referencing the social graph 1000, social-networking system 960may analyze the number and/or type of edges 1006 connecting particularuser nodes 1002 and concept nodes 1004 when calculating a coefficient.As an example and not by way of limitation, user nodes 1002 that areconnected by a spouse-type edge (representing that the two users aremarried) may be assigned a higher coefficient than a user nodes 1002that are connected by a friend-type edge. In other words, depending uponthe weights assigned to the actions and relationships for the particularuser, the overall affinity may be determined to be higher for contentabout the user's spouse than for content about the user's friend. Inparticular embodiments, the relationships a user has with another objectmay affect the weights and/or the ratings of the user's actions withrespect to calculating the coefficient for that object. As an exampleand not by way of limitation, if a user is tagged in first photo, butmerely likes a second photo, social-networking system 960 may determinethat the user has a higher coefficient with respect to the first photothan the second photo because having a tagged-in-type relationship withcontent may be assigned a higher weight and/or rating than having alike-type relationship with content. In particular embodiments,social-networking system 960 may calculate a coefficient for a firstuser based on the relationship one or more second users have with aparticular object. In other words, the connections and coefficientsother users have with an object may affect the first user's coefficientfor the object. As an example and not by way of limitation, if a firstuser is connected to or has a high coefficient for one or more secondusers, and those second users are connected to or have a highcoefficient for a particular object, social-networking system 960 maydetermine that the first user should also have a relatively highcoefficient for the particular object. In particular embodiments, thecoefficient may be based on the degree of separation between particularobjects. The lower coefficient may represent the decreasing likelihoodthat the first user will share an interest in content objects of theuser that is indirectly connected to the first user in the social graph1000. As an example and not by way of limitation, social-graph entitiesthat are closer in the social graph 1000 (i.e., fewer degrees ofseparation) may have a higher coefficient than entities that are furtherapart in the social graph 1000.

In particular embodiments, social-networking system 960 may calculate acoefficient based on location information. Objects that aregeographically closer to each other may be considered to be more relatedor of more interest to each other than more distant objects. Inparticular embodiments, the coefficient of a user towards a particularobject may be based on the proximity of the object's location to acurrent location associated with the user (or the location of a clientsystem 930 of the user). A first user may be more interested in otherusers or concepts that are closer to the first user. As an example andnot by way of limitation, if a user is one mile from an airport and twomiles from a gas station, social-networking system 960 may determinethat the user has a higher coefficient for the airport than the gasstation based on the proximity of the airport to the user.

In particular embodiments, social-networking system 960 may performparticular actions with respect to a user based on coefficientinformation. Coefficients may be used to predict whether a user willperform a particular action based on the user's interest in the action.A coefficient may be used when generating or presenting any type ofobjects to a user, such as advertisements, search results, news stories,media, messages, notifications, or other suitable objects. Thecoefficient may also be utilized to rank and order such objects, asappropriate. In this way, social-networking system 960 may provideinformation that is relevant to user's interests and currentcircumstances, increasing the likelihood that they will find suchinformation of interest. In particular embodiments, social-networkingsystem 960 may generate content based on coefficient information.Content objects may be provided or selected based on coefficientsspecific to a user. As an example and not by way of limitation, thecoefficient may be used to generate media for the user, where the usermay be presented with media for which the user has a high overallcoefficient with respect to the media object. As another example and notby way of limitation, the coefficient may be used to generateadvertisements for the user, where the user may be presented withadvertisements for which the user has a high overall coefficient withrespect to the advertised object. In particular embodiments,social-networking system 960 may generate search results based oncoefficient information. Search results for a particular user may bescored or ranked based on the coefficient associated with the searchresults with respect to the querying user. As an example and not by wayof limitation, search results corresponding to objects with highercoefficients may be ranked higher on a search-results page than resultscorresponding to objects having lower coefficients.

In particular embodiments, social-networking system 960 may calculate acoefficient in response to a request for a coefficient from a particularsystem or process. To predict the likely actions a user may take (or maybe the subject of) in a given situation, any process may request acalculated coefficient for a user. The request may also include a set ofweights to use for various factors used to calculate the coefficient.This request may come from a process running on the online socialnetwork, from a third-party system 970 (e.g., via an API or othercommunication channel), or from another suitable system. In response tothe request, social-networking system 960 may calculate the coefficient(or access the coefficient information if it has previously beencalculated and stored). In particular embodiments, social-networkingsystem 960 may measure an affinity with respect to a particular process.Different processes (both internal and external to the online socialnetwork) may request a coefficient for a particular object or set ofobjects. Social-networking system 960 may provide a measure of affinitythat is relevant to the particular process that requested the measure ofaffinity. In this way, each process receives a measure of affinity thatis tailored for the different context in which the process will use themeasure of affinity.

In connection with social-graph affinity and affinity coefficients,particular embodiments may utilize one or more systems, components,elements, functions, methods, operations, or steps disclosed in U.S.patent application Ser. No. 11/503,093, filed 11 Aug. 2006, U.S. patentapplication Ser. No. 12/977,027, filed 22 Dec. 2010, U.S. patentapplication Ser. No. 12/978,265, filed 23 Dec. 2010, and U.S. patentapplication Ser. No. 13/632,869, field 1 Oct. 2012, each of which isincorporated by reference.

In particular embodiments, one or more of the content objects of theonline social network may be associated with a privacy setting. Theprivacy settings (or “access settings”) for an object may be stored inany suitable manner, such as, for example, in association with theobject, in an index on an authorization server, in another suitablemanner, or any combination thereof. A privacy setting of an object mayspecify how the object (or particular information associated with anobject) can be accessed (e.g., viewed or shared) using the online socialnetwork. Where the privacy settings for an object allow a particularuser to access that object, the object may be described as being“visible” with respect to that user. As an example and not by way oflimitation, a user of the online social network may specify privacysettings for a user-profile page identify a set of users that may accessthe work experience information on the user-profile page, thus excludingother users from accessing the information. In particular embodiments,the privacy settings may specify a “blocked list” of users that shouldnot be allowed to access certain information associated with the object.In other words, the blocked list may specify one or more users orentities for which an object is not visible. As an example and not byway of limitation, a user may specify a set of users that may not accessphotos albums associated with the user, thus excluding those users fromaccessing the photo albums (while also possibly allowing certain usersnot within the set of users to access the photo albums). In particularembodiments, privacy settings may be associated with particularsocial-graph elements. Privacy settings of a social-graph element, suchas a node or an edge, may specify how the social-graph element,information associated with the social-graph element, or content objectsassociated with the social-graph element can be accessed using theonline social network. As an example and not by way of limitation, aparticular concept node 1004 corresponding to a particular photo mayhave a privacy setting specifying that the photo may only be accessed byusers tagged in the photo and their friends. In particular embodiments,privacy settings may allow users to opt in or opt out of having theiractions logged by social-networking system 960 or shared with othersystems (e.g., third-party system 970). In particular embodiments, theprivacy settings associated with an object may specify any suitablegranularity of permitted access or denial of access. As an example andnot by way of limitation, access or denial of access may be specifiedfor particular users (e.g., only me, my roommates, and my boss), userswithin a particular degrees-of-separation (e.g., friends, orfriends-of-friends), user groups (e.g., the gaming club, my family),user networks (e.g., employees of particular employers, students oralumni of particular university), all users (“public”), no users(“private”), users of third-party systems 970, particular applications(e.g., third-party applications, external websites), other suitableusers or entities, or any combination thereof. Although this disclosuredescribes using particular privacy settings in a particular manner, thisdisclosure contemplates using any suitable privacy settings in anysuitable manner.

In particular embodiments, one or more servers 962 may beauthorization/privacy servers for enforcing privacy settings. Inresponse to a request from a user (or other entity) for a particularobject stored in a data store 964, social-networking system 960 may senda request to the data store 964 for the object. The request may identifythe user associated with the request and may only be sent to the user(or a client system 930 of the user) if the authorization serverdetermines that the user is authorized to access the object based on theprivacy settings associated with the object. If the requesting user isnot authorized to access the object, the authorization server mayprevent the requested object from being retrieved from the data store964, or may prevent the requested object from be sent to the user. Inthe search query context, an object may only be generated as a searchresult if the querying user is authorized to access the object. In otherwords, the object must have a visibility that is visible to the queryinguser. If the object has a visibility that is not visible to the user,the object may be excluded from the search results. Although thisdisclosure describes enforcing privacy settings in a particular manner,this disclosure contemplates enforcing privacy settings in any suitablemanner.

FIG. 11 illustrates an example computer system 1100. In particularembodiments, one or more computer systems 1100 perform one or more stepsof one or more methods described or illustrated herein. In particularembodiments, one or more computer systems 1100 provide functionalitydescribed or illustrated herein. In particular embodiments, softwarerunning on one or more computer systems 1100 performs one or more stepsof one or more methods described or illustrated herein or providesfunctionality described or illustrated herein. Particular embodimentsinclude one or more portions of one or more computer systems 1100.Herein, reference to a computer system may encompass a computing device,and vice versa, where appropriate. Moreover, reference to a computersystem may encompass one or more computer systems, where appropriate.

This disclosure contemplates any suitable number of computer systems1100. This disclosure contemplates computer system 1100 taking anysuitable physical form. As example and not by way of limitation,computer system 1100 may be an embedded computer system, asystem-on-chip (SOC), a single-board computer system (SBC) (such as, forexample, a computer-on-module (COM) or system-on-module (SOM)), adesktop computer system, a laptop or notebook computer system, aninteractive kiosk, a mainframe, a mesh of computer systems, a mobiletelephone, a personal digital assistant (PDA), a server, a tabletcomputer system, or a combination of two or more of these. Whereappropriate, computer system 1100 may include one or more computersystems 1100; be unitary or distributed; span multiple locations; spanmultiple machines; span multiple data centers; or reside in a cloud,which may include one or more cloud components in one or more networks.Where appropriate, one or more computer systems 1100 may perform withoutsubstantial spatial or temporal limitation one or more steps of one ormore methods described or illustrated herein. As an example and not byway of limitation, one or more computer systems 1100 may perform in realtime or in batch mode one or more steps of one or more methods describedor illustrated herein. One or more computer systems 1100 may perform atdifferent times or at different locations one or more steps of one ormore methods described or illustrated herein, where appropriate.

In particular embodiments, computer system 1100 includes a processor1102, memory 1104, storage 1106, an input/output (I/O) interface 1108, acommunication interface 1110, and a bus 1112. Although this disclosuredescribes and illustrates a particular computer system having aparticular number of particular components in a particular arrangement,this disclosure contemplates any suitable computer system having anysuitable number of any suitable components in any suitable arrangement.

In particular embodiments, processor 1102 includes hardware forexecuting instructions, such as those making up a computer program. Asan example and not by way of limitation, to execute instructions,processor 1102 may retrieve (or fetch) the instructions from an internalregister, an internal cache, memory 1104, or storage 1106; decode andexecute them; and then write one or more results to an internalregister, an internal cache, memory 1104, or storage 1106. In particularembodiments, processor 1102 may include one or more internal caches fordata, instructions, or addresses. This disclosure contemplates processor1102 including any suitable number of any suitable internal caches,where appropriate. As an example and not by way of limitation, processor1102 may include one or more instruction caches, one or more datacaches, and one or more translation lookaside buffers (TLBs).Instructions in the instruction caches may be copies of instructions inmemory 1104 or storage 1106, and the instruction caches may speed upretrieval of those instructions by processor 1102. Data in the datacaches may be copies of data in memory 1104 or storage 1106 forinstructions executing at processor 1102 to operate on; the results ofprevious instructions executed at processor 1102 for access bysubsequent instructions executing at processor 1102 or for writing tomemory 1104 or storage 1106; or other suitable data. The data caches mayspeed up read or write operations by processor 1102. The TLBs may speedup virtual-address translation for processor 1102. In particularembodiments, processor 1102 may include one or more internal registersfor data, instructions, or addresses. This disclosure contemplatesprocessor 1102 including any suitable number of any suitable internalregisters, where appropriate. Where appropriate, processor 1102 mayinclude one or more arithmetic logic units (ALUs); be a multi-coreprocessor; or include one or more processors 1102. Although thisdisclosure describes and illustrates a particular processor, thisdisclosure contemplates any suitable processor.

In particular embodiments, memory 1104 includes main memory for storinginstructions for processor 1102 to execute or data for processor 1102 tooperate on. As an example and not by way of limitation, computer system1100 may load instructions from storage 1106 or another source (such as,for example, another computer system 1100) to memory 1104. Processor1102 may then load the instructions from memory 1104 to an internalregister or internal cache. To execute the instructions, processor 1102may retrieve the instructions from the internal register or internalcache and decode them. During or after execution of the instructions,processor 1102 may write one or more results (which may be intermediateor final results) to the internal register or internal cache. Processor1102 may then write one or more of those results to memory 1104. Inparticular embodiments, processor 1102 executes only instructions in oneor more internal registers or internal caches or in memory 1104 (asopposed to storage 1106 or elsewhere) and operates only on data in oneor more internal registers or internal caches or in memory 1104 (asopposed to storage 1106 or elsewhere). One or more memory buses (whichmay each include an address bus and a data bus) may couple processor1102 to memory 1104. Bus 1112 may include one or more memory buses, asdescribed below. In particular embodiments, one or more memorymanagement units (MMUs) reside between processor 1102 and memory 1104and facilitate accesses to memory 1104 requested by processor 1102. Inparticular embodiments, memory 1104 includes random access memory (RAM).This RAM may be volatile memory, where appropriate Where appropriate,this RAM may be dynamic RAM (DRAM) or static RAM (SRAM). Moreover, whereappropriate, this RAM may be single-ported or multi-ported RAM. Thisdisclosure contemplates any suitable RAM. Memory 1104 may include one ormore memories 1104, where appropriate. Although this disclosuredescribes and illustrates particular memory, this disclosurecontemplates any suitable memory.

In particular embodiments, storage 1106 includes mass storage for dataor instructions. As an example and not by way of limitation, storage1106 may include a hard disk drive (HDD), a floppy disk drive, flashmemory, an optical disc, a magneto-optical disc, magnetic tape, or aUniversal Serial Bus (USB) drive or a combination of two or more ofthese. Storage 1106 may include removable or non-removable (or fixed)media, where appropriate. Storage 1106 may be internal or external tocomputer system 1100, where appropriate. In particular embodiments,storage 1106 is non-volatile, solid-state memory. In particularembodiments, storage 1106 includes read-only memory (ROM). Whereappropriate, this ROM may be mask-programmed ROM, programmable ROM(PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM),electrically alterable ROM (EAROM), or flash memory or a combination oftwo or more of these. This disclosure contemplates mass storage 1106taking any suitable physical form. Storage 1106 may include one or morestorage control units facilitating communication between processor 1102and storage 1106, where appropriate. Where appropriate, storage 1106 mayinclude one or more storages 1106. Although this disclosure describesand illustrates particular storage, this disclosure contemplates anysuitable storage.

In particular embodiments, I/O interface 1108 includes hardware,software, or both, providing one or more interfaces for communicationbetween computer system 1100 and one or more I/O devices. Computersystem 1100 may include one or more of these I/O devices, whereappropriate. One or more of these I/O devices may enable communicationbetween a person and computer system 1100. As an example and not by wayof limitation, an I/O device may include a keyboard, keypad, microphone,monitor, mouse, printer, scanner, speaker, still camera, stylus, tablet,touch screen, trackball, video camera, another suitable I/O device or acombination of two or more of these. An I/O device may include one ormore sensors. This disclosure contemplates any suitable I/O devices andany suitable I/O interfaces 1108 for them. Where appropriate, I/Ointerface 1108 may include one or more device or software driversenabling processor 1102 to drive one or more of these I/O devices. I/Ointerface 1108 may include one or more I/O interfaces 1108, whereappropriate. Although this disclosure describes and illustrates aparticular I/O interface, this disclosure contemplates any suitable I/Ointerface.

In particular embodiments, communication interface 1110 includeshardware, software, or both providing one or more interfaces forcommunication (such as, for example, packet-based communication) betweencomputer system 1100 and one or more other computer systems 1100 or oneor more networks. As an example and not by way of limitation,communication interface 1110 may include a network interface controller(NIC) or network adapter for communicating with an Ethernet or otherwire-based network or a wireless NIC (WNIC) or wireless adapter forcommunicating with a wireless network, such as a WI-FI network. Thisdisclosure contemplates any suitable network and any suitablecommunication interface 1110 for it. As an example and not by way oflimitation, computer system 1100 may communicate with an ad hoc network,a personal area network (PAN), a local area network (LAN), a wide areanetwork (WAN), a metropolitan area network (MAN), or one or moreportions of the Internet or a combination of two or more of these. Oneor more portions of one or more of these networks may be wired orwireless. As an example, computer system 1100 may communicate with awireless PAN (WPAN) (such as, for example, a BLUETOOTH WPAN), a WI-FInetwork, a WI-MAX network, a cellular telephone network (such as, forexample, a Global System for Mobile Communications (GSM) network), orother suitable wireless network or a combination of two or more ofthese. Computer system 1100 may include any suitable communicationinterface 1110 for any of these networks, where appropriate.Communication interface 1110 may include one or more communicationinterfaces 1110, where appropriate. Although this disclosure describesand illustrates a particular communication interface, this disclosurecontemplates any suitable communication interface.

In particular embodiments, bus 1112 includes hardware, software, or bothcoupling components of computer system 1100 to each other. As an exampleand not by way of limitation, bus 1112 may include an AcceleratedGraphics Port (AGP) or other graphics bus, an Enhanced Industry StandardArchitecture (EISA) bus, a front-side bus (FSB), a HYPERTRANSPORT (HT)interconnect, an Industry Standard Architecture (ISA) bus, an INFINIBANDinterconnect, a low-pin-count (LPC) bus, a memory bus, a Micro ChannelArchitecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, aPCI-Express (PCIe) bus, a serial advanced technology attachment (SATA)bus, a Video Electronics Standards Association local (VLB) bus, oranother suitable bus or a combination of two or more of these. Bus 1112may include one or more buses 1112, where appropriate. Although thisdisclosure describes and illustrates a particular bus, this disclosurecontemplates any suitable bus or interconnect.

Herein, a computer-readable non-transitory storage medium or media mayinclude one or more semiconductor-based or other integrated circuits(ICs) (such, as for example, field-programmable gate arrays (FPGAs) orapplication-specific ICs (ASICs)), hard disk drives (HDDs), hybrid harddrives (HHDs), optical discs, optical disc drives (ODDs),magneto-optical discs, magneto-optical drives, floppy diskettes, floppydisk drives (FDDs), magnetic tapes, solid-state drives (SSDs),RAM-drives, SECURE DIGITAL cards or drives, any other suitablecomputer-readable non-transitory storage media, or any suitablecombination of two or more of these, where appropriate. Acomputer-readable non-transitory storage medium may be volatile,non-volatile, or a combination of volatile and non-volatile, whereappropriate.

Herein, “or” is inclusive and not exclusive, unless expressly indicatedotherwise or indicated otherwise by context. Therefore, herein, “A or B”means “A, B, or both,” unless expressly indicated otherwise or indicatedotherwise by context. Moreover, “and” is both joint and several, unlessexpressly indicated otherwise or indicated otherwise by context.Therefore, herein, “A and B” means “A and B, jointly or severally,”unless expressly indicated otherwise or indicated otherwise by context.

The scope of this disclosure encompasses all changes, substitutions,variations, alterations, and modifications to the example embodimentsdescribed or illustrated herein that a person having ordinary skill inthe art would comprehend. The scope of this disclosure is not limited tothe example embodiments described or illustrated herein. Moreover,although this disclosure describes and illustrates respectiveembodiments herein as including particular components, elements,functions, operations, or steps, any of these embodiments may includeany combination or permutation of any of the components, elements,functions, operations, or steps described or illustrated anywhere hereinthat a person having ordinary skill in the art would comprehend.Furthermore, reference in the appended claims to an apparatus or systemor a component of an apparatus or system being adapted to, arranged to,capable of, configured to, enabled to, operable to, or operative toperform a particular function encompasses that apparatus, system,component, whether or not it or that particular function is activated,turned on, or unlocked, as long as that apparatus, system, or componentis so adapted, arranged, capable, configured, enabled, operable, oroperative.

What is claimed is:
 1. A method comprising: by a computing device,analyzing an image to classify at least one expression displayed on aface shown in a captured image; by the computing device, selecting oneor more ideograms based on the at least one expression, wherein theselecting is based on one or more match scores for the at least oneexpression; and by the computing device, presenting, on a screenassociated with the computing device, the selected ideograms inassociation with the face.
 2. The method of claim 1, wherein theanalyzing the image to classify the at least one expression displayed onthe face comprises: detecting the face in the image; generating acharacterization of features of the face; and classifying the at leastone expression based on the characterization.
 3. The method of claim 1,wherein the face is associated with a user, and wherein the generating acharacterization of the features comprises: characterizing attributes ofthe features with respect to benchmark metrics for a particularexpression.
 4. The method of claim 3, wherein the benchmark metrics forthe features of the face associated with the user comprise acharacterization of attributes of the features when the particularexpression is displayed on the face.
 5. The method of claim 1, whereinthe face is associated with a user, and wherein the generating acharacterization of the features comprises: characterizing attributes ofthe features with respect to benchmark metrics for the features, whereinthe benchmark metrics are stored in association with a user profile forthe user.
 6. The method of claim 1, wherein the match scores aredetermined based on an ideogram dictionary or an ideogram usage history.7. The method of claim 6, wherein the ideogram dictionary or theideogram usage history is associated with the user, associated with oneor more social-networking connections of the user, associated with oneor more users sharing common attributes with the user, or based ongeneral statistics regarding trending ideograms.
 8. The method of claim1, wherein presenting the selected ideograms in association with theface comprises providing a user interface including the selectedideograms for insertion into a message.
 9. The method of claim 1,wherein presenting the selected ideograms in association with the facecomprises providing a user interface to post the selected ideograms forannotation of a content item.
 10. One or more computer-readablenon-transitory storage media embodying software that is operable whenexecuted to: analyze an image to classify at least one expressiondisplayed on a face shown in a captured image; select one or moreideograms based on the at least one expression, wherein the selecting isbased on one or more match scores for the at least one expression; andpresent, on a screen associated with the computing device, the selectedideograms in association with the face.
 11. The media of claim 10,wherein the software operable when executed to analyze the image toclassify the at least one expression displayed on the face comprisessoftware operable when executed to: detect the face in the image;generate a characterization of features of the face; and classify the atleast one expression based on the characterization.
 12. The media ofclaim 10, wherein the face is associated with a user, and wherein thesoftware operable when executed to generate a characterization of thefeatures comprises software operable when executed to: characterizeattributes of the features with respect to benchmark metrics for aparticular expression.
 13. The media of claim 12, wherein the benchmarkmetrics for the features of the face associated with the user comprise acharacterization of attributes of the features when the particularexpression is displayed on the face.
 14. The media of claim 10, whereinthe face is associated with a user, and wherein the software operablewhen executed to generate a characterization of the features comprisessoftware operable when executed to: characterize attributes of thefeatures with respect to benchmark metrics for the features, wherein thebenchmark metrics are stored in association with a user profile for theuser.
 15. A system comprising: one or more processors; and a memorycoupled to the processors comprising instructions executable by theprocessors, the processors being operable when executing theinstructions to: analyze an image to classify at least one expressiondisplayed on a face shown in a captured image; select one or moreideograms based on the at least one expression, wherein the selecting isbased on one or more match scores for the at least one expression; andpresent, on a screen associated with the computing device, the selectedideograms in association with the face.
 16. The system of claim 15,wherein the processors being operable when executing the instructions toanalyze the image to classify the at least one expression displayed onthe face comprises the processors being operable when executing theinstructions to: detect the face in the image; generate acharacterization of features of the face; and classify the at least oneexpression based on the characterization.
 17. The system of claim 15,wherein the match scores are determined based on an ideogram dictionaryor an ideogram usage history.
 18. The system of claim 17, wherein theideogram dictionary or the ideogram usage history is associated with theuser, associated with one or more social-networking connections of theuser, associated with one or more users sharing common attributes withthe user, or based on general statistics regarding trending ideograms.19. The system of claim 15, wherein the processors being operable whenexecuting the instructions to present the selected ideograms inassociation with the face comprises the processors being operable whenexecuting the instructions to provide a user interface including theselected ideograms for insertion into a message.
 20. The system of claim15, wherein the processors being operable when executing theinstructions to present the selected ideograms in association with theface comprises the processors being operable when executing theinstructions to provide a user interface to post the selected ideogramsfor annotation of a content item.